Bi re 


A Summary of the 
Research on the 
Effects of K-12 Test 
eee 
—. 2018 


CEO a: 423 


NCEO Report 423 


A Summary of the Research on the Effects 
of K-12 Test Accommodations: 2018 


Christopher M. Rogers, Sheryl S. Lazarus, and Martha L. Thurlow 


February 2021 


All rights reserved. Any or all portions of this document may be reproduced 
and distributed without prior permission, provided the source is cited as: 


Rogers, C. M., Lazarus, S. S., & Thurlow, M. L. (2021). A summary of the 
research on the effects of K-12 test accommodations: 2018 (NCEO Report 
423). National Center on Educational Outcomes. 


NCEO 


National Center on 
Educational Outcomes 


The Center is supported through Cooperative Agreements (#H326G160001) 
with the Research to Practice Division, Office of Special Education Programs, 
U.S. Department of Education. The Center is affiliated with the Institute on 
Community Integration at the College of Education and Human Develop- —_ US. Office of Special 


Education Programs 


ment, University of Minnesota. The contents of this report were developed 
under the Cooperative Agreement from the U.S. Department of Education, 
but does not necessarily represent the policy or opinions of the U.S. Depart- 
ment of Education or Offices within it. Readers should not assume endorse- 
ment by the federal government. 


Project Officer: David Egnor 


In collaboration with: fe th 
. = 
AEMe ccssore (MOE) Wested@ 
EOuFRORATION % hey £ 
NCEO Core Staff 
Sheryl S. Lazarus, Director Darrell Peterson 
Deb A. Albus Virginia Ressa 
Jessica Bowman Christopher Rogers 
Gail Ghere Jennifer Sommerness 
Linda Goldstone Kathy Strunk 
Andrew Hinkle Martha L. Thurlow 
Kristi K. Liu Terri Vandercook 
Charity Funfe Tatah Mentan Yi-Chen Wu 


Michael L. Moore 


National Center on Educational Outcomes 

University of Minnesota * 207 Pattee Hall 

150 Pillsbury Dr. SE * Minneapolis, MN 55455 

Phone 612/626-1530 * Fax 612/624-0879 

http://www.nceo.info 

The University of Minnesota shall provide equal access to and opportunity in its programs, facilities, and 


employment without regard to race, color, creed, religion, national origin, gender, age, marital status, disability, 
public assistance status, veteran status, sexual orientation, gender identity, or gender expression. 


This document is available in alternative formats upon request. 


Executive Summary 


The use of accommodations during assessments continues to be important for students with dis- 
abilities and for states as they establish and revise their accommodations policies. This importance 
is reflected in continued research to investigate the effects of accommodations. Key issues under 
continued investigation include how accommodations affect test scores, how educators and students 
perceive accommodations, and how accommodations are selected and implemented. 


The purpose of this report is to provide an update on the state of the research on testing accom- 
modations. Previous reports by the National Center on Educational Outcomes (NCEO) have cov- 
ered research published since 1999. In this report, we summarize the research published in 2018. 
During 2018, 11 research studies were published on the topic of testing accommodations in the 
U.S. elementary and secondary education system. 


Purpose of research: Over 80 percent of the research published in 2018 was to evaluate the ef- 
fects on test scores when K-12 assessments were administered with accommodations. The next 
most frequent purpose was to report on perceptions and preferences about accommodations use. 
The majority of studies (about 82%) addressed multiple purposes, and about half of the studies 
investigated both the performance impact and test-taker perceptions of accommodations. 


Research design: All studies reported primary data collection on the part of the researchers, 
rather than drawing only on existing data sets. Experimental or quasi-experimental designs were 
used in over 60 percent of the studies. Researchers also applied a variety of other quantitative and 
qualitative methodologies, including survey data analyses and interview protocols. 


Types of assessments, content areas: A wide variety of instrument types were used. Over two- 
fifths of the studies used academic content items drawn from specific sources outside of the study 
authors’ work, and almost one-fifth of studies used state criterion-referenced tests. Over four- 
fifths of the studies used non-academic protocols or surveys developed by the study authors. A 
few studies (27%) used norm-referenced measures. Nearly all of the studies (82%) used multiple 
types of data. Mathematics and reading were the most common K-12 content areas addressed in 
the research published in 2018. Other content areas included science and English language arts. 
One-fifth of all studies addressed more than one content area. 


Participants: Participants were most frequently students, spanning a range of grades throughout 
K-12 education, including the elementary, middle, and high school levels. Over one-third of stud- 
ies had participants in more than one school level. Some studies included educators or students’ 
parents as participants. Studies varied in the number of participants; some studies included fewer 
than 10 participants, while over half of the studies included more than 30 participants, and one 
study had nearly one thousand. 


Disability categories: Visual impairment including blindness was the most common disability 
category of participants in the research published in 2018, accounting for more than one-third of 
the studies. Learning disabilities comprised the next most commonly studied category. Attention- 
related disabilities, intellectual disabilities, and hearing impairment including deafness were 
included in one quarter of the studies altogether. 


Accommodations: Presentation accommodations were the most frequently studied category of 
accommodations. Electronic administration was the most-studied individual accommodation. A 
relatively large proportion of the studies published in 2018 reported on accommodations (e.g., 
calculator, sign language administration) that were investigated in only one study each. 


Findings: Nine studies analyzed the effect of accommodations. Of these nine, the accommoda- 
tion studied most often was electronic administration pertaining to how test-takers are presented 
assessments as well as how they respond to assessments, and whether additional equipment is 
required for assessment participation. Other findings on the effects of accommodations were re- 
ported by only one study each. Seven studies provided findings on effects of accommodations on 
math assessments, three studies presented findings on effects for reading assessments, and two 
studies analyzed the effects of accommodations in more than one content area. Some studies pro- 
vided comparisons of different versions of accommodations rather than focusing on the effects of 
a single accommodation. In many studies, accommodations benefited at least some students with 
disabilities, yet had no effect on the performance of other students with disabilities. No studies 
indicated a negative effect of accommodations for students with disabilities. 


Almost two-thirds of the studies provided findings about perceptions of accommodations. Over 
one quarter of K-12 accommodations studies published in 2018 provided findings about ac- 
commodations use patterns by students or implementation practices by educators. Many studies 
provided insights about students’ general impressions about accommodations as well as their 
preferences among accommodations. Most students with disabilities had positive perceptions of 
accommodations. Relatively few findings were available on educators’ and parents’ perceptions 
about accommodations. 
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Overview 


Access to assessments for students with disabilities is supported through the use of accom- 
modations. With accommodations, students with disabilities, including English learners with 
disabilities, are better able to show their academic knowledge and skills. Accommodations also 
enable these students to participate in state assessments, as required by the Individuals with Dis- 
abilities Education Act (IDEA) of 2004 and by the 2015 reauthorization of the Elementary and 
Secondary Education Act (ESEA). Accommodations are changes in materials and procedures 
that do not compromise the validity of assessment results and interpretations of those results. 
Evidence is needed to ensure that validity is not compromised. It is also important to examine 
perceptions of accommodations and implementation issues because these influence whether 
accommodations are used appropriately. Research conducted on accommodations can provide 
states with information useful for policy on accommodations. 


To synthesize accommodations research efforts completed across the years, the National Cen- 
ter on Educational Outcomes (NCEO) has published a series of reports on accommodations 
research. The time periods included 1999-2001 (Thompson et al., 2002), 2002—2004 (John- 
stone et al., 2006), 2005-2006 (Zenisky & Sireci, 2007), 2007-2008 (Cormier et al., 2010), 
2009-2010 (Rogers et al., 2012), 2011—2012 (Rogers et al., 2014), 2013-2014 (Rogers et al., 
2016), 2015-2016 (Rogers et al., 2019), and 2017 (Rogers et al., 2020). The report summarizing 
the 2017 empirical studies narrowed the focus to K-12 research in the United States context. 


The purpose of this report is to present a synthesis of the research on test accommodations for 
U.S. elementary and secondary students (K—12) published in 2018. The academic literature 
described here encompasses empirical studies of performance comparability, investigations into 
accommodations use, implementation practices, and perceptions of the effectiveness of accom- 
modations. As a whole, the current research body offers a broad view and a deep examination 
of issues pertaining to assessment accommodations. Reporting the findings of recent research 
studies was the collective goal of these analyses. 


Review Process 


Similar to the process used in NCEO’s past accommodations research syntheses, a number of 
sources were accessed to complete the review of the K-12 accommodations research published 
in 2018. Specifically, five research databases were consulted: Educational Resources Information 
Center (ERIC), PsycINFO, Academic Search Premier, Digital Dissertations, and Educational 
Abstracts. To help confirm the thoroughness of our searches, we used the web search engine 
Google Scholar to locate additional research. In addition, a hand-search of at least 50 journals 
was completed in efforts to ensure that no qualifying study was missed. A list of hand-searched 
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journals is available on the NCEO website (https://nceo.info/Resources/bibliographies/accom- 


modations/methods-for-identifying). 


Online archives of several organizations also were searched for relevant publications. These 
organizations included the following: Behavioral Research and Teaching (BRT) at the Univer- 
sity of Oregon (https://www.brtprojects.org/publications/); the College Board Research Library 
(http://research.collegeboard.org); the National Center for Research on Evaluation, Standards, 
and Student Testing (CRESST; http://cresst.org/education/); and the Wisconsin Center for Edu- 
cational Research (WCER; https://www.weer.wisc.edu/publications). 


The initial search was completed in December, 2018. A second search was completed in April 
2019 to ensure that all articles published in 2018 were found and included in this review. Within 
each of these research databases and publications archives, we used a sequence of search terms. 
Terms searched for this review were: 

¢ standardized (also large-scale, state, standards-based) test (also testing) changes 

¢ standardized (also large-scale, state, standards-based) test (also testing) modification(s) 

¢ standardized (also large-scale, state, standards-based) test (also testing) 

* accommodation(s) 

* test changes 

* test modifications 

* test accommodations 


Many of these search terms were used as delimiters when searches yielded large pools of docu- 
ments found to be irrelevant to the searches. 


The research documents from these searches were then considered for inclusion in this review 
using several criteria. 


1. This analysis included only research published or defended (in doctoral dissertations) in 2018. 


2. The scope of the research was limited to investigations of accommodations for regular as- 
sessments; hence, articles specific to accommodations for alternate assessments, accommoda- 
tions for instruction or learning, and universal design in general were not part of this review. 


3. Research involving English learners was included only if the target population was English 
learners with disabilities. 


4. Presentations from professional conferences were not searched or included in this review, 
based on NCEO’s criterion to include only research that would be accessible to readers and 
had gone through the level of peer review typically required for publication in professional 
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journals or through a doctoral committee review. (This criterion was implemented for the 
first time during the 2007—2008 review.) 


5. To be included in the online bibliography and summarized in this report, studies needed to 
involve (a) experimental manipulation of an accommodation; (b) investigation of the com- 
parability of test scores across accommodated and non-accommodated conditions, or across 
more than one accommodated condition; or (c) examination of survey results or interview 
data sets about students’ or teachers’ knowledge or perceptions of accommodations. 


6. This report was focused on research on students in United States schools; consequently, 
studies with only participants in other national contexts were not included. 


7. The current report includes only research pertaining to the primary and secondary levels of 
the education system, that is, from Kindergarten through grade 12. 


8. This report does not include literature reviews or meta-analyses (unlike previous NCEO 
accommodations research reports). 


These limitations do not necessarily apply to NCEO’s Accommodations for Students with Dis- 
abilities Bibliography, which is an online database (https://nceo.info/Resources/bibliographies/ 
accommodations/bibliography). The Accommodations Bibliography continues to include re- 


search in non-US. settings. Also, postsecondary accommodations research is included in the 
Bibliography. Many literature reviews of various kinds also are included in the Bibliography. 


To reflect the wide range of accommodations research in the K-12 system that was published 
in 2018, the studies are examined and summarized for the following features: (a) publication 
type, (b) purposes of research, (c) research type and data collection source, (d) assessment or 
data collection focus, (e) characteristics of the independent and dependent variables under study, 
and (f) comparability of findings between studies in similar domains. 


Results 


Publication Type 


A total of 11 studies was published between January 2018 and December 2018. As shown in 
Figure 1, nine of the 11 studies were journal articles, and two were dissertations. None were 
published professional reports released by research organizations or entities (e.g., WCER). 


The total number of studies published on accommodations in the K—12 educational context 
in 2018 (N=11) decreased from 2017, when it was 14, yet was similar to previous years. In 
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NCEO’s previous report (Rogers et al., 2020), we applied our more stringent inclusion crite- 
ria for analyzing the research literature. The numbers of published K—12 studies were nine in 
2015 and nine in 2016. The numbers of journal articles increased, from four in 2015, to seven 
in 2016, to seven in 2017, to nine in 2018. The largest variability in type was for dissertations, 
with five in 2015, two in 2016, seven in 2017, and two in 2018. The absence of reports from 
research organizations has continued, with zero in 2015, 2016, 2017, and 2018. This (2018) 
review included nine journal articles from seven different journals, including three articles from 
one academic journal. In previous years’ data, there were seven journal articles from seven 
different journals in 2017 (Rogers et al., 2020), seven articles from seven different journals in 
2016, and four articles from four different journals in 2015 (Rogers et al., 2019). Appendix A 
presents information about the publication type of each study published in 2018. 


Figure 1. Percentage of K-12 Accommodations Studies in 2018 by Publication Type 


0% Dissertations 
18% 


Journal Articles 
82% 


Purposes of the Research 


Several purposes were identified in the K-12 accommodations research published in 2018. 
Table 1 shows the primary focuses of the 11 studies included in this review. Two studies each 
listed a single purpose (see Appendix B). The majority of studies sought to accomplish multiple 
purposes. In these cases, we identified the “primary purpose” based on the title of the work or 
the first-mentioned purpose in the text. 
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Table 1. Primary Purpose of K-12 Studies in 2018 


p Number of | Percent of 
i dd Studies Studies 


Discern effects on assessment scores 
only students with disabilities (6 studies; 55% of studies) pe fm 
only students without disabilities (2 studies; 18% of studies) 
both students with and without disabilities (0 studies; 0% of studies) 


Report on implementation practices and accommodations use 


[Discuss issues HT 
[Review or summarize research on testaccommodations | | 0% 
[Comparetestitems TH 
Evaluateteststructure TH 
jinvestigatetestvalidity TH 


The most frequent primary purpose for research published during 2018 was discerning the 
effects of accommodations through score comparisons during test performance. Two studies’ 
primary focus was on accommodations implementation practices or accommodations use. Only 
one study focused primarily on the perceptions of accommodations. 


Table 2 shows the multiple purposes investigated by nine of the 11 studies, as well as the singular 
purpose of two studies. Although primary purposes were limited to three (effects, implementation 
or use, and perceptions), many studies had more than one purpose. Nine studies analyze, either 
as a primary or other purpose, the effects of accommodations through comparing performance 
data for students only with disabilities (n=7) and for students only without disabilities (n=2). 
The next most widely studied purpose (n=7) was examinations of the perceptions of students, 
teachers, or students’ parents on accommodations. Three studies described implementation 
practices or incidence of use during assessment. Two studies each summarized relevant accom- 
modations research or discussed accommodations-related issues. 


Appendix B presents the studies’ many purposes. Nearly all of the studies (n=9) addressed two 
or more purposes. Six studies (Bone & Bouck, 2018; Bouck et al., 2018; Ha & Fang, 2018; 
Hansen et al., 2018; Kong et al., 2018; McLaughlin & Kamei-Hannan, 2018) had two purposes, 
with almost all (except McLaughlin & Kamei-Hannan) including analysis of effects of accom- 
modations and examining participants’ perceptions. Two studies (Cobb, 2018; Pangatungan, 
2018) investigated three purposes. Beal and Rosenblum (2018) addressed four purposes, in- 
cluding effects of accommodations, perceptions and preferences between accommodations, 
implementation and use, and discussing accommodations-related issues. 
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Table 2. All Purposes of K-12 Studies in 2018 


Purpose Number of | Percent of 
ia Studies Studies 


Discern effects on assessment scores 
only students with disabilities (7 studies; 64% of studies) are 
only students without disabilities (2 studies; 18% of studies) 
both students with and without disabilities (0 studies; 0% of studies) 


|Report on implementation practices and accommodationsuse_ || 27% 
|Study/compare perceptions and preferences aboutuse | 7 | 4% 
[Discuss issues 18% 
[Review or summarize research on testaccommodations | 2 | 18% 
[Comparetestitems TH 
Evaluateteststructure HT 
jinvestigatetestvalidity TH 


Note. Of the 11 studies reviewed for this report, 9 reported addressing two or more purposes. Thus, the number 
of studies totals more than 11, and more than 100%. 


Research Type and Data Collection Source 


Quasi-experimental design was the most frequent type of accommodations research published 
in 2018, comprising nearly one-half of the 11 K-12 studies. As shown in Table 3, the research- 
ers for all quasi-experimental studies, and in fact, all 11 studies, gathered the data themselves. 
One of the studies (Cobb, 2018) also included secondary data in an analysis of the high school’s 
annually-collected state assessment performance scores. Three studies were descriptive quantita- 
tive designs, and two studies reported using experimental designs. Only one study used primarily 
or exclusively qualitative data. No studies published in 2018 used correlational or longitudinal 
designs, so those designs are not included in Table 3. 


Table 3. Research Type and Data Collection Source for K-12 Studies in 2018 


Research Design Data Collection Source Total 
Primary Secondary [eas 
Quasi-experimental 5 0 5 
Descriptive quantitative 3 0' 3 
Experimental 2 0 2 
Descriptive qualitative 1 0 1 
Totals 11 0 11 


' One study collected primary data, yet also examined extant state assessment data. 


We observed a difference in the number of studies published in 2018 that used primary data 
collection sources when compared to those that used secondary data collection sources. All 11 


6 NCEO 


studies used primary sources and none used only secondary sources. In studies published in 
2017, we found that over 20 percent of the studies employed secondary data sources. (Appendix 
A presents research designs and data collection sources for individual studies.) 


Data Collection Methods and Instruments 


Research published in 2018 used the methods shown in Figure 2 to collect study data. Seventy- 
three percent of the studies (n=8) used performance data acquired through academic content 
testing. In some of the cases (e.g., Hansen et al., 2018), tests were administered as part of the 
study, while in one case (Cobb, 2018), an extant data source was used. Observations comprised 
a large proportion (55%) of studies’ data, most commonly task or test completion times (n=4, 
36%). In contrast, observation data comprised about 14 percent of the 2017 studies (Rogers et 
al., 2020). Other frequent data sources in 2018 K—12 studies include surveys (n=6, 55%) and 
interviews (n=4, 36%). Studies did not report academic grade records as relevant data. Nine 
studies (82%) reported using more than one method or tool to gather data. See Appendix A for 
additional details about each study’s data collection methods. 


Figure 2. Data Collection Methods Used in K-12 Studies in 2018 


Number of Studies 


Test 8 
oO Observations 6 
{o} 
<= 
a) 
oO 
= Survey 6 
Interview Protocol 4 


Note. Of the 11 studies reviewed for this report, 5 reported using two data collection methods and 4 reported us- 
ing three data collection methods. Thus, the number of methods in this figure totals more than 11. 
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All of the studies published in 2018 used some type of data collection instrument (see Table 

4). The instruments were placed into seven categories based on the sources of the instruments: 

¢ Non-academic protocols or surveys developed by study authors 

¢ Surveys or academic tests developed by education professionals or drawn by researchers 
from other sources 

¢ State criterion-referenced academic assessments 

¢ Norm-referenced academic achievement measures 

¢ Norm-referenced cognitive ability measures 

* Non-state criterion-referenced academic assessments 

¢ Other 


In 10 studies, non-academic protocols developed by the authors of the studies were used. This 
was the most commonly-used type of instruments. Examples included questionnaires with rat- 
ing scales of social validity checks on the students’ testing experiences (Bone & Bouck, 2018), 
interview protocols for uncovering students’ perceptions about ASL clarity as well as prefer- 
ences between signer versions (Hansen et al., 2017), and observation protocols for measuring 
reading speed and accuracy (McLaughlin & Kamei-Hannan, 2018). 


Surveys or academic tests developed by researchers or other education professionals using 
sources outside of the study were employed in seven studies. An example of a survey in the 
studies in 2018 was adapted by the researchers (Ha & Fang, 2018) from the USE Questionnaire 
(Lund, 2001) containing elements of usability: usefulness, satisfaction, and ease of use. An 
example of an academic test that was created drew mathematics practice items for preparing 
for a standardized math assessment, linked to academic standards expected of a student at the 
middle school level or early high school level (Hansen et al., 2018). 


Table 4. Data Collection Instrument Sources for K-12 Studies in 2018 


instrument Soureels Number of | Percent of 
: $ ¢ Studies Studies 
Non-academic surveys, interview protocols, and observation protocols 5 
10 91% 
developed by study author/s 
Surveys or academic tests developed by professionals or researchers ‘i 
; 7 64% 
using sources outside of current study 
State criterion-referenced assessments 2 18% 
Norm-referenced academic achievement measures 2 18% 
Norm-referenced cognitive ability measures 2 18% 
Criterion-referenced (non-state) academic achievement measures 0 0% 
Other? 3 27% 


Note. Nine studies (82%) used instruments from more than one source; therefore, numbers total more than the 
11 studies represented, and percentages total more than 100. 


@ Other: see Appendix C, Table C-1 for specific information in Bara et al., 2018; Bouck et al., 2018; and Kelly et 
al., 2018. 
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Table 4. Data Collection Instrument Sources for K-12 Studies in 2018 (continued) 
KEY 


Surveys sets of items of an attitudinal or self-report nature 
Tests either course- or classroom-based 
Assessments statewide or large-scale assessments 


Protocols sets of questions, usually presented in an interview format 
Measures norm-referenced academic achievement or cognitive ability instruments 


State criterion-referenced assessments included the extant data set from the 2017 Smarter Bal- 
anced Assessment Consortium (SBAC) English language arts and mathematics assessments 
administered in California (Cobb, 2018), as well as 40 publicly-available items from already- 
completed test booklets for the 2013 Texas Assessment of Academic Readiness grade 3 reading 
achievement test (Kelly et al., 2018). Two norm-referenced academic achievement measures 
were used for pre-screening or independent checking of performance, including the Kaufman 
Test of Educational Achievement, Second Edition (KTEA™-II; Kaufman & Kaufman, 2004) 
and the Johns Basic Reading Inventory (Johns, 2012). Three norm-referenced cognitive ability 
measures were each used in one study: the Key Math-3 Diagnostic Assessment (KeyMath-3; 
Connolly, 2007), the Wechsler Intelligence Scale for Children, Fourth Edition (WISC, 4th ed.; 
Wechsler, 2004), and the Purdue Spatial Visualization Test: Revised (PSVT-R; Yoon, 2011). 
Criterion-referenced academic achievement measures were not utilized in the 2018 studies. 
Over 80 percent of all studies (n=9) used instrumentation of more than one kind. We present a 
complete listing of instruments used in each of the studies in Table C-1 in Appendix C, includ- 
ing the related studies or other sources for these instruments, when available. 


Content Area Assessed 


Nine studies published during 2018 focused on accommodations used in specific academic 
content areas (see Appendix C, Table C-2, for additional details about the content areas). As 
shown in Table 5, mathematics was the most commonly studied content area. Table 5 was 
constructed, in part, by applying the same criteria used in the current report retroactively to the 
data from the 2015—2016 report (Rogers et al., 2019), as well as drawing data from the 2017 
report (Rogers et al., 2020). In three of the four years of our accommodations research literature 
reviews (all except 2016), reading and mathematics were the most common content areas for 
accommodations research, yet have varied in terms of which of the two was the most common 
content area in any particular year. 


Cumulatively, science has tended to be the third most frequent content area, with 9 total studies 
across the four years of research reviews. In 2018, about one-fifth of studies examined accom- 
modations impact data for more than one content area. The inclusion of multiple content area 
analyses varied in frequency across the four years, from none in 2016 to four (36%) in 2017. 
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The remaining academic content areas—writing, “other language arts,” and social studies— 
comprised similar proportions of the research across the four years of studies. 


Table 5. Academic Content Area Assessed in K-12 Studies across Three Reports 


Content Area Assessed 2015? 2016 2017° 2018° 
Mathematics 3 (33%) 1 (14%) 5 (45%) 7 (64%) 
Reading 3 (33%) 1 (14%) 6 (55%) 5 (46%) 
Writing 2 (22%) 1 (14%) 1 (9%) 0 (0%) 
Other language arts? 1 (11%) 0 (0%) 2 (18%) 1 (9%) 
Science 1 (11%) 4 (57%) 3 (27%) 1 (9%) 
Social studies 1 (11%) 0 (0%) 0 (0%) 0 (0%) 
Multiple content? 2 (22%) 0 (0%) 4 (36%) 2 (18%) 
Total (of Relevant Studies) 9 7 11 11 


a Studies in 2015 included studies that addressed two content areas. 


> Studies in 2017 included studies that addressed more than one content area (i.e., two content areas, three 
content areas). 


° Studies in 2018 included studies that addressed more than one content area (i.e., two content areas, three 
content areas). 


4 Detailed descriptions of what constituted “Other Language Arts” for the 2018 studies can be found in Appendix 
C, Table C-2. 


® Because some studies investigated effects in more than one content area, the percentages total more than 100. 
Research Participants 


The studies in this examination of accommodations research published in 2018 included par- 
ticipants in several roles (see Figure 3 and Appendix D). A majority of the studies included only 
students (n=8, 73%). Only educators participated in one study (Cobb, 2018). The remaining stud- 
ies had combinations of participant populations. Students with visual impairments and teachers 
participated in the Beal and Rosenblum (2018) study that examined effects of, and perspectives 
on use and practices related to, electronic tablet-based assessments. Pangatungan (2018) sought 
the viewpoints of the parents and teachers of students with and without learning disabilities on 
the students’ use of oral delivery for supporting academic performance and behavior. 


Table 6 details the size and composition of the participant groups in the K-12 research studies 
published during 2018. For additional detail by study, see Appendix D. The size of the partici- 
pant groups varied from three participants (Bara et al., 2018; Bouck et al., 2018; McLaughlin & 
Kamei-Hannan, 2018) to 964 (Kong et al., 2018). The largest single set of studies published in 
2018 involved only students with disabilities (5 studies); in comparison, only two studies (Ha 
& Fang, 2018; Kong et al., 2018) had participants who were only students without disabilities. 
One study (Kelly et al., 2018) compared groups of students with and without disabilities, yet 
no studies directly compared data from an identical number of students from both populations. 
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Figure 3. Types of Research Participants for K-12 Studies in 2018 
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Two studies engaged only educators (Cobb, 2018) or educators and parents (Pangatungan, 2018) 
as participants, and one study (Beal & Rosenblum, 2018) had both educators and students as 
participants. 


Table 6. Participant Sample Sizes and Ratio of K-12 Students with Disabilities in 2018 


Number of Research Number of Studies by Proportion of Sample Com- 
Participants by Study prising Individuals with Disabilities 
0-24% | 25-49% | 50-74% | 75—-100% | Total 
1-9 0 0 0 4 4 
10-24 1 0 0 0 1 
25-49 2 0 0 1 3 
50-99 0 0 1 1 2 
100—1000 1 0 0 0 1 
Total 3 0 1 7 
School Level 


Students in the elementary, middle, and high school levels participated in nine of the 11 studies 
published in 2018 (see Table 7; also see Appendix D for students’ specific grade levels when 
available). Most studies included students in more than one grade; only two studies (Bara et 
al., 2018; Bone & Bouck, 2018) each examined student data for a single grade level. Four 
studies involved students from more than one school level: one study (Kelly et al., 2018) had 
elementary and middle school participants, one study (McLaughlin & Kamei-Hannan, 2018) 
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had middle and high school participants, one study (Hansen et al., 2018) had high school and 
some postsecondary students, and one study (Beal & Rosenblum, 2018) had participants from 
all three school levels. 


Table 7. School Level of Research Participants for K-12 Studies in 2018 


Education Level of Participants in Studies Number of Studies Percent of Studies 
Elementary school (K—5) 3 27% 
Middle school (6—8) 6 55% 
High school (9-12) 4 37% 
Not applicable 2 18% 


Note. Four studies (37%) had participants in more than one education level; therefore, the numbers total more 
than the 11 studies represented, and percentages total more than 100. 


Disability Categories 


The K-12 accommodations research published in 2018 addressed a number of disability cat- 
egories (see Appendix D for details). As shown in Table 8, two studies included only students 
without disabilities, and one study included both students with and without disabilities; two 
studies did not include students in the sample, and one study included both students and non- 
students in the sample. Of the nine studies including students with disabilities, the most com- 
monly studied student disability category was visual impairment including blindness (n=4). Two 
studies (Bara et al., 2018; McLaughlin & Kamei-Hannan, 2018) included only participants with 
visual impairments. One study (Kelly et al., 2018) compared students with visual impairments 
to students without disabilities. One study (Beal & Rosenblum, 2018) included students with 
visual impairments and their teachers. 


In addition to visual impairments, one other study included students from only one disability 
category. Specifically, Hansen and colleagues (2018) gathered data from students with hearing 
impairments including deafness who used American Sign Language in the educational setting. 
Two studies (Bone & Bouck, 2018; Bouck et al., 2018) included students from two disability 
categories. The Bone and Bouck study analyzed data from students with learning disabilities 
and those with learning disabilities plus attention problems. 


Types of Accommodations 
The specific types of accommodations included in the research published in 2018 are summa- 


rized in Table 9. A complete listing of accommodations examined in each study is provided in 
Appendix E, Table E-1, and by accommodation type in Tables E-2 through E-6. 
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Table 8. Disabilities Reported for Research Participants for K-12 Studies in 2018 


Disabilities of Research Participants Number of Studies Percent of Studies 
Visual impairment/blindness 4 36% 
Learning disabilities 2 18% 
Attention problem 1 9% 
Hearing impairment/Deafness 1 9% 
Intellectual disabilities 1 9% 
No disability 3 27% 
Not applicable (non-students) 3 27% 


Note. Some studies had participants who fit into more than one disability category; therefore, the numbers total 
more than the 11 studies represented, and percentages total more than 100. 


Presentation was the most commonly investigated accommodation category, in seven studies. 
Equipment/materials accommodations were addressed in five studies, and response accommoda- 
tions were examined in four studies. Much less frequently-studied was setting, in two studies. 
Some studies (n=4) included accommodations from more than one category. Of those, three 
studies included accommodations from each of three accommodations types, and one study 
included accommodations from two accommodations types. 


The most broadly studied presentation accommodation (see Appendix E, Table E-2) was oral 
delivery, in three studies (Bara et al., 2018; Hansen et al., 2018; Pangatungan, 2018). In previ- 
ous NCEO accommodations research reports, we merged the data for the three ways that this 
accommodation was provided: (a) by a test administrator live and in-person, (b) with a recorded 
human voice, and (c) as simulated speech via text-to-speech devices or software. Oral delivery 
presented live and in-person was combined with two-dimensional illustrations and tactile illus- 
trations supports in one study (Bara et al.). Recorded video delivery of American Sign Language 
(ASL) by a person, and ASL simulated by an avatar through software, were compared in one 
study (Hansen et al.). One study (Pangatungan) described perceptions of live, in-person, oral 
delivery and text-to-speech software. 


Details on equipment accommodations are presented in Appendix E, Table E-3. Electronic 
administration was investigated in three studies (Beal & Rosenblum, 2018; Kong et al., 2018; 
McLaughlin & Kamei-Hannan, 2018). This accommodation encompasses three types of ac- 
commodation: (a) how students are presented tests, (b) computer or electronic tablet equipment, 
and (c) the way students record their test item responses. Two studies (Beal & Rosenblum, 
McLaughlin & Kamei-Hannan) investigated the effects of being presented tests on paper or 
electronically: one study addressed effects on reading speed (McLaughlin & Kamei-Hannan), 
and the other combined large print or braille with either paper or electronic tablet to discern 
any effects on performance (Beal & Rosenblum). In one study (Kong et al., 2018), researchers 
explicitly compared administration by computer to administration by electronic tablet, in order 
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to discern whether students’ use of keyboards or touchscreens for recording their responses af- 
fected test item response speed. 


The equipment/materials accommodation of technological aids was the focus of two studies 
(Bouck et al., 2018; Ha & Fang, 2018), in addition to electronic administration (Beal & Rosen- 
blum, Kong et al., McLaughlin & Kamei-Hannan). 


Details on response accommodations are presented in Appendix E, Table E-4. The response ac- 
commodation of calculator was researched in one study (Bone & Bouck, 2018). Three response 
accommodations studies focused on electronic administration studies. 


Setting accommodations were investigated in one study (see Appendix E, Table E-5). Kelly and 
colleagues (2018) examined individual administration. 


No studies published in the 11 2018 studies examined timing or scheduling accommodations. 
In previous years, a common timing or scheduling accommodation was extended time. 


Table 9. Accommodations in Reviewed Research in 2018 


Accommodations Category Number of Studies 
Equipment/Materials 


Timing/Scheduling —= ee 
Setting 


Note. Five studies investigated accommodations from more than one category; therefore, the numbers in this 
table total more than the 11 studies represented. 


Research Findings 


The findings of the studies on accommodations published in 2018 are summarized here accord- 
ing to results of the studies. Appendix F presents details on individual studies. These findings 
were consistent with the stated purposes and focuses of the studies. The findings included sets 
of research about specific accommodations, such as tactile graphics. Other studies examined 
impacts of aggregated sets of accommodations sometimes called “bundles.” We also identify 
findings on the impact of unique accommodations—those examined in only one study—such 
as scientific calculators, and touch screen or keyboard response. We report on perceptions of 
accommodations, including those of student test-takers, educators, and students’ parents. We 
describe implementation conditions as well as patterns of use of various accommodations. This 
report also presents findings by academic content areas: math, reading and other language arts, 
and science. 
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Impact of Accommodations 


Research published in 2018 that examined the effects of accommodations on assessment 
performance for K-12 students with disabilities totaled 9 studies (see Figure 4; see also Ap- 
pendix F for details about each of these studies). We report here the effects of three discrete 
accommodations, including tactile graphics (illustrations or manipulatives), oral delivery—live 
and in-person, as well as oral delivery with simulated voice via text-to-speech software—and 
electronic administration. See Appendix F for further details on accommodations with only one 
associated finding. 


Figure 4. Effects of Specific Accommodations for K-12 Studies in 2018 
Number of Studies 


Electronic administration 


Tactile graphics (incl. 
manipulatives) 


Calculator (scientific) 
Keyboard 
Sign language administration 


Touch screen 


Specific Accommodations 


Not specified 


Note. Five studies each examined the separate impacts of more than one accommodation; therefore, the total 
exceeds the number of studies represented (9). 


The two most investigated accommodations in 2018 were investigated in three studies each: 
electronic administration (Beal & Rosenblum, 2018; Kong et al., 2018; McLaughlin & Kamei- 
Hannan, 2018) and tactile graphics (Bara et al., 2018; Bouck et al., 2018; Ha & Fang, 2018). 
Electronic administration was provided in comparison with standard paper format (Beal & 
Rosenblum, McLaughlin & Kamei-Hannan) or comparing computer with keyboard respond- 
ing against electronic tablet with touchscreen responding (Kong et al.). Tactile graphics were 
presented both as low-tech physical manipulatives and (high-tech) electronic tablet virtual 
manipulatives (Bouck et al., Ha & Fang), as well as comparing two-dimensional illustration 
against three-dimensional paper-based illustrations and objects (Bara et al.). 


Beal and Rosenblum (2018) found that most students (88%) with visual impairments across 
grade levels demonstrated significantly higher scores on math test items when using the test 
administered on electronic tablet, in comparison with paper; only five percent of these stu- 
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dents scored higher with paper-based tests. McLaughlin and Kamei-Hannan (2018) found that 
middle and high school students with visual impairments had mixed accuracy results between 
paper and electronic tablet formats, with about 10 percent difference or less in error incidence, 
concluding that such errors had no significant effect on comprehension. However, the study’s 
participants tended to read text moderately faster when using electronic tablets over when using 
paper. Further, across repeated sessions, students each tended to increase their reading speeds 
by five to 10 words per minute with the tablet, and decreased speed with paper. The researchers 
indicated that the participants’ increased reading speed was an important improvement in flu- 
ency. Although not comparing test scores, Kong and colleagues (2018) found that high school 
students without disabilities took significantly longer on average to complete science, reading, 
and math tests presented on tablets with touchscreen responding than on computer screens with 
keyboard responding. In specific analyses of item types, including traditional multiple-choice 
items and technologically-enhanced items such as fill-in-the-blank, they found little variation 
in the response time pattern, with small item effect sizes; the exception was that drag-and-drop 
items took essentially the same amount of time. 


Bara and colleagues (2018) found that grade 1 students with visual impairments differed in 
their recall of narrative reading details, comparing the two-dimensional illustrations and the 
three-dimensional (tactile) images conditions: while all three students recalled more with il- 
lustrations than without, two children correctly recalled about 25 to 35 percent more details in 
the 3-D images condition, and the other child recalled slightly more details with the 2-D im- 
ages. Of the types of story details, students recalled only object transformations better with 3-D 
images. The other related studies investigated the impact for middle school students of using 
manipulatives over not using them when responding to fraction-based math problems. Bouck and 
colleagues (2018) found that students with intellectual or learning disabilities performed better 
with manipulatives than without them, but that both the concrete or physical manipulatives and 
the virtual manipulatives presented on an electronic screen resulted in similar scores. Further, 
students took longer to complete the math tasks when using manipulatives than when not having 
them, yet they took similar amounts of time between the physical and virtual manipulatives. 
Ha and Fang (2018) found that study participants, who had no reported disabilities, scored over 
20 percent higher on spatial skills with the manipulatives than without them. When comparing 
student performance by gender, male students scored about 23 percent higher on average than 
female students on the paper format (without the manipulatives tool); when both groups used 
the manipulatives tool, their scores significantly improved, yet the gender-based average score 
difference had decreased, with males scoring six percentage points higher than females. 


We identified separate reportable findings on the impact of four unique accommodations—that 
is, each of these accommodations that were the focus of just one study. Effects of these four 
unique accommodations were examined by three studies: 

¢ calculator (Bone & Bouck, 2018) 
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¢ keyboard and touch screen (Kong et al., 2018) 
* sign language administration (Hansen et al., 2018) 


One study (Cobb, 2018) described the impact of students with various disabilities using unspeci- 
fied accommodations naturalistically (as assigned in their individualized education program/IEP 
plans) when completing state assessments. Results indicated that students seemed to improve 
in their English language arts scores but not their mathematics scores. 


Findings for each of these unique accommodations are reported in Appendix F. 
Perceptions about Accommodations 


Seven studies (Beal & Rosenblum, 2018; Bone & Bouck, 2018; Bouck et al., 2018; Ha & 
Fang, 2018; Hansen et al., 2018; Kong et al., 2018; Pangatungan, 2018) provided findings on 
perceptions of accommodations (see Appendix F for details on these studies). Nearly all of 
them reported on students’ perceptions, although one study (Pangatungan) highlighted only the 
perceptions of students’ parents and educators. In addition to students’ perceptions, Beal and 
Rosenblum also detailed the perceptions of educators. Two studies (Beal & Rosenblum, Bone 
& Bouck) yielded students’ perceptions about specific accommodations, when compared to 
conditions of not using accommodations. The other four studies on student perceptions (Bouck 
et al., Ha & Fang, Hansen et al., Kong et al.) detailed both perceptions as well as preferences. 


Students had positive perceptions of most of the accommodations. Students with visual impair- 
ments in grades 4-10 mostly (78%) expressed positive experiences with the tablet app, rather 
than reading math items on paper, while 15 percent preferred the paper-based format reportedly 
due to higher familiarity (Beal & Rosenblum, 2018). Similarly, students with learning disabili- 
ties, or both learning disabilities and attention difficulties, indicated liking the opportunity to 
use scientific calculators, and preferred using this accommodation over not having it available 
(Bone & Bouck, 2018). Students in grades 7—8 with learning disabilities or intellectual dis- 
abilities enjoyed both the concrete and virtual manipulatives over not using them (Bouck et al., 
2018). Middle school students (without disabilities) expressed via survey questions using a 7-8 
point scale mostly positive perceptions about the virtual and physical manipulatives. Students’ 
median responses were highest on ease of use and ease of learning to use, both rating a ‘6,’ or 
‘agree.’ The median response on usefulness and satisfaction were both rated as ‘somewhat agree,’ 
or a ‘5’ (Ha & Fang, 2018). High school students and other youth with hearing impairments 
indicated their perceptions of the quality of the signing by the human: 60 percent of survey re- 
spondents endorsed either ‘very good’ or ‘excellent,’ with only one participant finding fault with 
the relatively fewer than desired pauses by the human when signing. In contrast, 10 percent of 
survey respondents rated the avatar’s simulated signing in one of the two highest ratings, with 


NCEO 17 


common criticisms including that the avatar’s facial expressions, mouthing of the words, and 
body movement were not sufficiently communicative (Hansen et al., 2018). 


Students also indicated preferences between versions of some accommodations. Bouck and col- 
leagues (2018) reported that two students preferred using the virtual manipulatives app while 
one had no preference. Ha and Fang (2018) found that most students (72%) preferred using both 
physical and virtual manipulatives, with students commenting that they thought the different 
manipulative types complemented each other, in that the physical touch connected them to the 
academic content, yet the computer-based manipulative quickly provided precise information 
about angles. Ha and Fang noted that most of the remaining students (19%) preferred using 
the virtual manipulative. Hansen and colleagues (2018) indicated that 29 of the 31 students in 
their study preferred the human presenting ASL. These researchers noted further that nearly all 
participants indicated their impressions that the English version of the math test would have 
been sufficient for them to answer the items correctly, and students suggested various ideas 
for improving the ASL supports for the math test. Unfortunately, Kong and colleagues (2018) 
presented very little student survey data from high school students without disabilities, limiting 
details about perceptions or preferences. 


Other non-student participants indicated similarly positive perceptions in general, highlighting 
specific considerations. Teacher surveys indicated students had higher motivation when using 
tablets over paper, and interviews yielded that more teachers (78%) had the impression that 
students were more engaged with the test on the tablet, while 22 percent reported no difference 
in engagement with the test on the tablet than on paper (Beal & Rosenblum, 2018). Pangatun- 
gan (2018), a dissertation researcher, presented substantial detail from surveys and interviews 
of both parents and teachers of students with and without learning disabilities in grades 4—5. 
Deriving five themes and 12 sub-themes, the researcher found that oral delivery was supportive 
of students’ reading performance, including school grades and state tests, according to teachers. 
Teachers also perceived that their reading aloud to students can model and instruct decoding, 
while preventing students from becoming discouraged. Parents also indicated that their children 
need and benefit from having access to various content including academic content through 
either the parents reading to them or students using computer-based oral delivery of content 
when completing schoolwork at home. 


Implementation and Use of Accommodations 


Three studies (Beal & Rosenblum, 2018; Cobb, 2018; Kelly et al., 2018) had findings related 
to accommodations use and implementation issues. In two studies (Beal & Rosenblum, 2018; 
Kelly et al., 2018), researchers described patterns of accommodations use, while one study 
(Cobb, 2018) provided information about educators’ accommodations implementation practices. 
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Beal and Rosenblum (2018) found that teachers of students with visual impairments indicated 
that these students seemed to need little help overall, and only for very few math test items, 
when using the electronic tablet format. Students typically sought assistance with navigation 
on the tablet, explaining the math problems, computing the math, and discussing the graphics. 
There were no apparent differences in numbers of items or degree of need for these assistance 
requests between the braille-using students and those reading print. Kelly and colleagues (2018) 
reported, from group-wise comparisons, that both groups of students with visual impairments— 
school-aged children with amblyopia and children with non-amblyopic strabismus—took about 
28 percent longer on average than their peers without visual impairments to complete the task 
of copying item responses from test booklets to answer forms. All three groups had similar rates 
of copying errors. The researchers expressed concern that students with visual impairments who 
were not blind, particularly those with substantially different vision in one eye over the other, 
can still have fine motor impairments. These students have demonstrable needs for, and benefits 
from, extended time or other accommodations, especially in high-stakes academic activities. 


Cobb (2018) found that all educator participants, both teachers and administrators, responded 
similarly on a survey describing accommodations implementation issues. Mean rating responses 
to specific survey items, on a 3-point scale, ranged from 2.3 to 2.6, showing that the high school 
was implementing important elements of an inclusive education system, such as teachers hav- 
ing substantial knowledge of accommodations (with a rating of 2.3) and availing all students 
of accommodations during instruction (rating 2.6). Open-ended question responses were about 
55 percent positive and 45 percent negative in terms of considering the needs of students with 
disabilities within the general education setting. Cobb concluded that, while accommodations, 
as a component of inclusive practices, were not flawlessly implemented in the first two years, 
the 2017 performance of the high school’s students with disabilities on the state ELA assessment 
increased to higher than the state average, after previously being below average, demonstrat- 
ing a higher than (state) average rate of improvement. Cobb also noted that the students with 
disabilities at the high school had not improved in state math assessment performance during 
the same time frame, persisting in low mean performance that remained below the statewide 
math assessment score average. 


Accommodations by Academic Content Assessments 


As in previous reports, we analyzed research findings according to the academic content area 
included in each of the studies. The content areas, presented in terms of the number of studies 
including them were: mathematics (n=7), reading (n=5), science (n=1), and other language arts 
(n=1). For each content area, we examined the impact on assessment performance, perceptions 
about accommodations, and implementation and use patterns. This analysis included all 11 
studies. One study (Kong et al., 2018) incorporated accommodations’ effects on three content 
areas: math, reading, and science assessments. Another study (Cobb, 2018) addressed imple- 
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mentation of accommodations in two areas, math and other language arts. (See Appendix F for 
a more detailed explanation of the findings of each study.) 


Mathematics. All seven of the studies involving mathematics presented findings on the effects 
of accommodations on performance. Four studies (Beal & Rosenblum, 2018; Bone & Bouck, 
2018; Bouck et al., 2018; Hansen et al., 2018) provided effects data only for students with 
disabilities, while two studies (Ha & Fang, 2018; Kong et al., 2018) did so only for students 
without disabilities. One study (Cobb, 2018) reported a secondary data analysis on extant data 
sets of math (and ELA) state assessment mean performance of students with disabilities. None 
of the math-related studies provided comparisons across groups of students with and without 
disabilities. 


Two studies using math assessments examined the effects of manipulatives, both concrete or 
tactile tools as well as high-tech manipulatives presented on screens (Bouck et al., 2018; Ha & 
Fang, 2018), while two other studies presented the impact of electronic administration (Beal & 
Rosenblum, 2018; Kong et al., 2018). One study reported on the impact of scientific calcula- 
tors (Bone & Bouck, 2018) and one study reported on the impact of ASL (Hansen et al., 2018). 


Accommodations benefited the math performance of students with disabilities in three studies 
(Beal & Rosenblum, 2018; Bone & Bouck, 2018; Bouck et al., 2018), and students without 
disabilities benefited from using supports in one study (Ha & Fang, 2018). Beal and Rosenblum 
(2018) found that students with visual impairments in grades 4-10 scored significantly higher 
on pre-algebra story problems that were presented on electronic tablets than when they were 
presented on paper. Both electronic and paper formats provided either enlarged font or braille 
tools, based on how each participant typically read tests. Bone and Bouck (2018) found that all 
participants—grade 8 students with learning disabilities, attention problems, or both—scored 
higher on multi-step math computation items when using scientific calculators than when they 
did not. The participants also tended to complete more task steps when using calculators. Bouck 
and colleagues (2018) indicated that grade 7—8 students with learning or intellectual disabilities 
had higher scores on fraction-related math problems when using manipulatives than when not 
using manipulatives, yet the scores were essentially the same between the concrete manipula- 
tives and the virtual manipulatives presented electronically. However, the students required more 
time to complete the math tasks when using manipulatives of either type than when not using 
them. Ha and Fang (2018) found that middle school students, who had no reported disabilities, 
scored higher on a norm-referenced spatial skill test (PSVT-R; cf. Yoon, 2011) associated with 
math reasoning when using a virtual and physical manipulatives tool than when completing the 
paper-based test without the tool. Further, the pattern of gender-linked baseline performance 
difference appeared to be nearly neutralized with the use of the manipulatives tool. 
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One study (Hansen et al., 2018) indicated that there was no significant difference between two 
versions of ASL; that is, high school students and other youth with hearing impairments who 
used ASL scored similarly on math items presented with the recorded human signing and the 
avatar-simulated signing. Finally, a study that did not measure score effects produced a negative 
finding on a different effect, between two versions of an accommodation: Kong and colleagues 
(2018) found that response times for high school students without disabilities on math (alge- 
bra) assessments were much higher, by 52 seconds for 10 items, on tablets with touchscreen 
responding than on computer screens with keyboard responding. 


Six of the seven math-related studies—besides Cobb (2018)— yielded findings about perceptions 

and preferences regarding accommodations. All six studies (Beal & Rosenblum, 2018; Bone 
& Bouck, 2018; Bouck et al., 2018; Ha & Fang, 2018; Hansen et al., 2018; Kong et al., 2018) 
gathered data from students, and one of these studies (Beal & Rosenblum, 2018) also described 
teachers’ perceptions of accommodations. Students’ positive views were reported in five studies 
(Beal & Rosenblum, 2018; Bone & Bouck, 2018; Bouck et al., 2018; Ha & Fang, 2018; Hansen 
et al., 2018), and their preferences for one version of the accommodations over another were 
also noted in three studies (Bouck et al., 2018; Ha & Fang, 2018; Hansen et al., 2018). 


Beal and Rosenblum (2018) found that students with visual impairments in grades 4-10 mostly 
(78%) expressed positive experiences with electronic administration, and 15 percent of them 
indicated that they would rather use the standard paper-based format. Beal and Rosenblum found 
that 78 percent of teachers had the impression that students were more engaged when taking 
the math test on the electronic tablet than on paper, and 22 percent perceived no difference in 
engagement between these formats. Bone and Bouck (2018) concluded that the students with 
disabilities in grade 8 increased in their positive views of using scientific calculators from the 
beginning to the end of the study. Bouck and colleagues (2018) found that all three students in 
grades 7-8 with learning or intellectual disabilities viewed manipulatives positively, with two 
preferring the virtual, screen-based manipulatives, and one having no preference. Ha and Fang 
(2018) concluded that middle school students with no reported disabilities gave relatively high 
median ratings for virtual and physical manipulatives on usability surveys: ‘6’ (on a 7-point 
scale) on ease of use and ease of learning, and ‘5’ on usefulness and satisfaction. Most students 
(72%) preferred using both types of manipulatives in combination, and 19 percent preferred 
using virtual manipulatives. Hansen and colleagues (2018) found that about 60 percent of 
high school students and other youth with hearing impairments thought the ASL presented by 
a recorded person was either “very good” or “excellent,” but that 10 percent rated ASL by an 
avatar in the same way. In fact, 29 of the 31 survey respondents preferred the human version, 
one preferred the avatar, and one had no preference. The avatar was reported not to adequately 
communicate ASL through facial expressions, mouthing of the words, and body movement; 
students’ additional suggestions were also detailed. Kong and colleagues (2018) gathered high 
school students’ survey data, yet presented very little detail on their perceptions or preferences. 
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Cobb (2018) concluded, from results of an educator survey (cf., Stetson & Associates, Inc., 
2014) in a specific high school, that the implementation of inclusive education policies and 
practices, including accommodations in instruction and assessment, did not seem to influence 
any increased performance on state math assessments for students with disabilities. 


Reading. Of the four studies involving reading, three studies (Bara et al., 2018; Kong et al., 
2018; McLaughlin & Kamei-Hannan, 2018) presented findings on the effects of accommoda- 
tions on assessment performance. Two studies (Bara et al., McLaughlin & Kamei-Hannan) 
provided effects data only for students with disabilities, and one study (Kong et al.) presented 
findings on the effects on response times for students with no reported disabilities, as well as the 
students’ perceptions of responding to two electronic administration versions. The remaining 
study (Pangatungan, 2018) described the perceptions of teachers and parents of students with 
and without disabilities about oral delivery of reading content. None of the reading-related stud- 
ies provided performance comparisons across groups of students with and without disabilities. 


Accommodations benefited the reading assessment performance in both studies for students 
with disabilities; participants in both studies had visual impairments. Examining data from three 
grade | students with visual impairments, Bara and colleagues (2018) found that two children 
correctly recalled about 25 to 35 percent more details in the 3-D images condition than in either 
the 2-D images or no-illustration conditions. The other child recalled slightly more details with 
the 2-D images over the other conditions. Further, the children recalled the narrative story’s 
elements differently: more object transformation details were recalled when using 3-D illustra- 
tions, while character-related details and other information were recalled at similar rates across 
the test conditions. McLaughlin and Kamei-Hannan (2018) reported mixed accuracy results for 
middle and high school students with visual impairments when using enlarged font presented 
on standard paper test format and presented on electronic tablet format, with about 10 percent 
difference or less in error incidence. The researchers concluding that such errors had no sig- 
nificant effect on reading comprehension. However, the study’s participants tended to read text 
moderately faster when using electronic tablets in comparison to when using paper. In fact, after 
repeated tablet use, students each tended to increase their reading speeds by five to 10 words 
per minute. The researchers concluded that electronic administration facilitated participants’ 
increased reading speed, a distinct improvement in fluency. 


Pangatungan (2018) indicated that teachers of grade 4—5 students with and without learning 
disabilities perceived that their reading aloud to students can model and instruct decoding, while 
preventing students from becoming discouraged. Parents of these same students indicated that 
their children need and benefit from having access to academic content through either the parents 
reading to them or students using computer-based oral delivery of content when completing 
schoolwork at home. The researcher concluded that oral delivery was supportive of students’ 
reading performance, including school grades and state tests, as well as reading homework. 


22 NCEO 


Science. Only one study (Kong et al., 2018) provided findings for science. The finding simul- 
taneously pertained to the comparison of two electronic administration response conditions 
during math and reading assessments, while not reporting details for the student perceptions of 
these accommodations. Kong and colleagues (2018) found that response times for high school 
students without disabilities on science assessments were much higher on tablets with touch- 
screen responding than on computer screens with keyboard responding. Further, this pattern 
was essentially the same across various test item types. 


Other Language Arts. Only one study (Cobb, 2018) provided findings for English language 
arts (ELA). Cobb (2018) examined data from educators on the implementation of inclusive 
education policies and practices including accommodations in instruction and assessment at a 
specific high school, and completed a secondary data analysis on extant data sets of ELA (and 
math) state assessment mean performance of students with disabilities. The researcher concluded 
that, although accommodations were not flawlessly implemented in the first two years, the 2017 
performance of the high school’s students with disabilities on the state ELA assessment increased 
to higher than the state average, after previously being below average, demonstrating a higher 
than (state) average rate of improvement. 


Discussion 


This report is the second consecutive NCEO report that has provided a snapshot of accommo- 
dations research literature involving only K—12 students published in a single year. This report 
summarizes accommodations research published in 2018. This is a narrower focus than most 
previous NCEO accommodations research reports (Cormier et al., 2010; Johnstone et al., 2006; 
Thompson et al., 2002; Rogers et al., 2012; Rogers et al., 2014; Rogers et al., 2016; Rogers et 
al., 2019; Zenisky et al., 2007); those incorporated more than one year of research literature, and 
encompassed elementary, secondary, and postsecondary education settings. We highlighted the 
types of accommodations that were studied, the purposes of the research, the research type, data 
sources, characteristics of the independent and dependent variables under study, and comparabil- 
ity of findings between studies in similar domains, including by specific accommodations and 
their performance effects. We also examined findings by academic content area. 


Mathematics and reading were the content areas most frequently addressed in the studies included 
in this analysis—together comprising nearly all of the studies that used content assessments. 
Science comprised more than one-tenth of the studies with content assessments, or almost one- 
tenth of all studies described in this report. These proportions are generally similar to those noted 
in previous NCEO accommodations research reports (Cormier et al., 2010; Johnstone et al., 
2006; Thompson et al., 2002; Rogers et al., 2012; Rogers et al., 2014; Rogers et al., 2016; Rog- 
ers et al., 2019; Rogers et al., 2020; Zenisky et al., 2007). An exception is that math comprised 
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a larger proportion of studies, while reading was addressed in a smaller proportion. Science 
continues to be a content area of interest in K-12 accommodations research. Students were the 
participant group in more than three quarters of the studies. Students with visual impairments 
including blindness were participants in about one-third of the studies reported, while students 
with learning disabilities participated in a smaller proportion of accommodations research in 
2018 than has been typical in recent years. Other disability categories receiving attention by 
some studies included attention-related disabilities, deafness/hearing impairment, and intel- 
lectual disabilities; each of these categories comprised one study each. 


Accommodations research has continued to generate substantial research activity and has ex- 
panded in breadth and depth. The number of studies we located increased across the span of 
NCEO’s accommodations research summary reports; for instance, in 2011—2012, there were 
49 identified studies, in 2013-2014, there were 53 studies, and in 2015—2016, there were 58 
studies. A substantial proportion of the studies examined in those previous reports highlighted 
research on the academic accommodations provided at institutions of higher education. Keep- 
ing in mind that the most recent NCEO report (Rogers et al., 2020) in the series addressed only 
one year, 2017, and was narrowed to the context of the U.S. K-12 school system, the 14 studies 
described were consistent with a continually increasing trend for this topic in research literature. 
Further, the current report also addressed one year of accommodations research (published in 
2018) within the U.S. K-12 context; it includes 11 studies, a number generally similar to the 
previous one-year U.S. K-12 accommodations research. 


Researchers have continued to explore a wide range of topics related to assessment accommo- 
dations, including the comparison of the effects of differing versions of accommodations, such 
as providing physical or virtual manipulatives, and human-signed or simulated ASL. Studies 
also continued to address multiple purposes and to examine various types of data. Along with 
investigating effects on content tests, researchers inquired about perceptions of students with 
disabilities and educators about accommodations, and students’ use patterns and educators’ 
implementation practices surrounding accommodations. 


Both quantitative analyses and thematic exploration of qualitative data have yielded findings 
that can inform and improve considerations for addressing students’ needs. For instance, Beal 
and Rosenblum (2018) investigated potential differences between traditional paper-based tests 
and electronically-administered tests for students with visual impairments, and discussed is- 
sues related to motivation and persistence during test-taking. Similarly, Pangatungan (2018) 
inquired about the perceptions of the parents and teachers of elementary students with learning 
disabilities, seeking to gain deeper understanding from important others about student engage- 
ment and motivation. These issue-driven inquiries can offer additional insights and perspectives 
about the provision of assessment accommodations to students with disabilities who need them. 
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Interest in research on accommodations continues. This research can inform states as they 
consider their accommodations policies. It can also provide information for others interested 
in assessment accommodations. 
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Appendix A 


Research Characteristics for K-12 Studies in 2018 


Publication Research F peta vel Collection Instru- 
Authors Research Design lection 
Type Type ment 
Source 
Bara et al. Journal Quantitative Descriptive Quantitative | Primary Observations 
eat els Journal Mixed Quasi-experimental Primary heen ney. 
blum Test 
Bone & Bouck | Journal Quantitative Experimental Primary Obselvallone, 
Survey, Test 
Bouck et al. Journal Mixed Experimental Primary Interviews Opeet 
vations, Test 
Cobb Dissertation | Quantitative Descriptive Quantitative | Primary’ Survey, Test 
Ha & Fang Journal Mixed Quasi-experimental Primary Survey, Test 
Hansen et al. Journal Mixed Quasi-experimental Primary Interview, Test 
Kelly et al. Journal Quantitative Descriptive Quantitative | Primary Observations, Test 
Kong et al. Journal Mixed Quasi-experimental Primary Observations; 
Survey, Test 
McLaughlin & Journal Quantitative Quasi-experimental Primary Observations, Test 
Kamei-Hannan 
Pangatungan Dissertation | Qualitative Descriptive Qualitative Primary lntepyiew OBSet: 


vations, Survey 


‘The researcher gathered primary data, yet also employed secondary (extant) state assessment data from the 
high school and the statewide assessment scores for analyses. 
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Research Purposes for K-12 Studies in 2018 
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Appendix C: Instrument Characteristics for K-12 Studies in 2018 


Table C-1. Specific Instrument Sources 


Authors 
Bara et al. 


Beal & Rosenblum 


Bone & Bouck 


NCEO 


Number 
of 
Instrument Sources and Descriptions Sources 


Author (observation): Numbers of verbal student-initiated interactions 
with the instructor, numbers of correct details recalled, and percentages 
of matches between text elements and illustrations, as measured by the 
researcher team from video-recordings. 


Author (interview, survey): Teacher survey rating the degrees of support 
that their students sought while completing math items, and the degrees 
of students’ motivation. Phone interview protocol with students and 

their teachers to report on their experiences and preferences, and 
observations, regarding the accommodations. 


Researcher (test): Sets of 24 mathematics story problems (also called 
word problems) aligned to Common Core State Standards for grade 6 
math, incorporating pre-algebra content, were completed by student 
participants under the two task completion conditions. 


Author (survey, observation): 9-item student survey that explored the 
social validity of using calculators both at the beginning and end of the 
study. Participants’ use or non-use of calculators when provided was 
observed and documented by researchers. 


Researcher (test): Mathematics problems based on Common 

Core State Standards were presented in five testing sessions; 

math computation items comprised the tests during baseline and 
intervention phases, then math word problems were given to check 
for skill generalization. Numbers of correct answers and numbers of 
items attempted were documented; partial credit was given for partial 
completion of items (such as evidence of steps of task completed). 
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Authors 


Bouck et al. 


Ha & Fang 
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Number 
of 
Instrument Sources and Descriptions Sources 


Author (observation, interview): Assessment task completion time 
was measured with a stopwatch app, observed and documented by 
researchers. Post-assessment interview questions pertaining to social 
validity, including students’ preferred manipulative type. 


Researcher (test): Sets of math tests with five items on the addition of 
fractions with unlike denominators comprised the accuracy measure; 
performance score was the number of correct answers out of five 
questions. Task independence was measured based on researchers’ 
observations of the proportion of total task steps completed without 
seeking or requiring prompting by test administrators, as documented 
on system of least prompts forms. 


Norm-ref Ach: (for screening purposes) Kaufman Test of Educational 
Achievement, Second Edition (KTEA™-II; Kaufman & Kaufman, 2004). 


Norm-ref Ability: (for screening purposes) Relevant subtests of Key 
Math-3 Diagnostic Assessment (KeyMath-3; Connolly, 2007). Wechsler 
Intelligence Scale for Children, Fourth Edition (WISC, 4th ed.; Wechsler, 
2004). 


Other: (for screening purposes) Vineland Adaptive Behavior Scales, 
Second Edition (Vineland-lIl; Sparrow, Cicchetti, & Balla, 2005). 


Researcher (survey): A questionnaire (Stetson & Associates, Inc., 2014) 
adapted by the researcher into a 31-item rating survey was completed 
by teachers and school administrators for rating the degree to which 
accommodations, part of a total of 11 inclusionary practices, were 
implemented in a California high school. Educators also reported their 
demographics and responded to a subset of open-ended questions in 
the survey. 


State Test: The extant data set of English language arts and 
mathematics assessments, the 2017 Smarter Balanced Assessment 
Consortium (SBAC) California assessment, from the high school was 
compared to the statewide assessment results. 


Author (survey): Student survey on preferences among the 
manipulative conditions and reasons for the ratings. 


Researcher (survey): Student survey of perspectives on Usefulness, 
Satisfaction, and Ease of Use (USE; Lund, 2001). 


Norm-ref Ability: Student participants’ scores on the Purdue Spatial 
Visualization Test: Revised (PSVT-R; Yoon, 2011), a 30-item multiple- 
choice assessment, were analyzed. The researchers noted that the 
spatial skill of mental rotation is associated with STEM (science, 
technology, and math) reasoning. 


NCEO 


Kelly et al. 


Kong et al. 


McLaughlin & 
Kamei-Hannan 


NCEO 


Number 
of 
Instrument Sources and Descriptions Sources 


Author (interview): A series of interview questions on student 
participants’ experience completing the test items using different ASL 
signers (human vs. avatar), including their perceptions about clarity of 
the signing, as well as their preferences for either signer version, and 
their suggestions for improving usability. 


Researcher (test): Mathematics test items were drawn from a set of 
practice items for preparing for a standardized math test, and were 
linked to standards expected of a student at the middle school or early 
high school level. Three math test items included graphics. 


Author (observation): Participants’ completion times were observed and | 3 
documented by researchers using a stopwatch. 


State Test: The accuracy of students’ test-taking behavior—transferring 
40 already answered correct responses of the 2013 Texas Assessment 
of Academic Readiness grade 3 reading achievement test (publicly 
available version) from the test booklet to a blank Scantron “bubble 
sheet”’—was measured based on the researchers’ comparisons of 

the participants’ transcribed answer forms to the actual provided item 
answers. 


Other: (for screening purposes) Participants’ visual impairments were 
prescreened for stereoacuity with the Randot Preschool Stereoacuity 
Test (Birch & The Retina Foundation of the Southwest, 2018) and 
Random Dot Stereo Butterfly test (Stereo Optical Company, Inc., 2018). 


Author (observation, survey): Student participants’ item response times 
were observed and documented by researchers. 10-question student 
survey on their familiarity with devices and on their perceptions of test- 
taking during the study. 


Researcher (test): Test items for (high school level) English II, Biology, 
and Algebra | were presented as traditional multiple-choice items and 
technologically-enhanced items: multiple select, drag and drop, fill in the 
blank, graph point, hot spot, and inline choice (from drop-down box). 
[Participants’ correctness scores were calculated for solution behavior 
and rapid-guessing behavior, not as primary data for comparison, but 
rather, for the purpose of checking for validity of these different test- 
taking behaviors.] 


Author (observation): Researchers documented observations, tracking 
completion time associated with reading speeds (in words per minute), 
as well as accuracy, with number of errors in pronunciation. 


Norm-ref Ach: The Johns Basic Reading Inventory (Johns, 2012) was 
used to independently check the reading skill levels for the participants. 
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Authors 
Pangatungan 


KEY: 


Number 
of 
Instrument Sources and Descriptions Sources 


Author (interview, survey, observation): Perceptions of parents and 
teachers were captured through a semi-structured interview protocol, 


documented in the researcher’s notes, with full quotes validated by 
interviewees. A survey rating the relative impact of oral delivery on 
students’ overall reading performance was also completed by the 
parents and teachers of students with and without learning disabilities 
and other difficulties. Field notes comprising the researcher’s 
observations served to authenticate the other data sources. 


Instrument Sources Type Abbreviations 
( 


Non-Academic Protocols or Surveys Developed by Study Author (Interview / 


Author/s 
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Observation/Survey) 


Researcher Test 


38. °2~2~CO NEE 


Table C-2. Content Areas Assessed 


McLaughlin & Kamei-Hannan 
: 


Note: This table encompasses the studies (N=11) which used assessments or tests on academic content area/s 
or cognitive skills. 


Social Studies 


4|n this study, other LA = identified by Smarter Balanced Assessment as English language arts, with both reading 
comprehension of literary and informational texts, and writing—producing effective and well-grounded writing. 
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Appendix D 


Participant and Sample Characteristics for K-12 Studies in 2018 


Disability 
Percent of Grade / Categories 
Unit of Sample with Education Included in 
Authors Analysis Sample Size Disabilities Level Sample 


Students [3 [100% 


Beal & Students; 73 59% grades 4-10 

Rosenblum Educators (n=43); 
teachers 
(n=30) 


Bone & Bouck | Students [5 100% LD; LD &AP 
Bouck etal. | Students 100% grades 7-8 ID, LD 


Cobb Educators 36 0% educators 
(teachers and 
others, at high 
school) 


HI 


Hansen etal. | Students 31 100% high school 
(n=12); 
postsecondary 
| (n=19) 
Kelly et al. Students 85 76% 8-6-12.1 years | VI, None 
| (grades 3-7) 
Kongetal. [Students [964.0% | high school 
mel 
; 


McLaughlin & | Students 100% middle to high | 
Kamei-Hannan school 
Pangatungan | Educators; 2 0% teachers and |N/A 
Parents parents (of 
students in 
grades 4-5) 
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Appendix E 


Accommodations Studied for K-12 Studies in 2018 


Table E-1. All Accommodations by Study 


Bouck etal. 
Cobb 
Ha & Fang Technological aid (physical/tactile and virtual manipulatives) (EQUIPMENT) 

Hansen eta. 


Wiet avian ManeacHannan Electronic administration, Large print/magnification, Font/background contrast 
g (PRESENTATION, EQUIPMENT, RESPONSE) 


Oral delivery, live/in-person; Oral delivery, text-to-speech (PRESENTATION) 


Author/s 


Le 
c 
co) 
© 
£ 

— 
— 
c 

= 
roe 
® 
D 
i 
© 

_ 


i 
fe} 
= 
© 
oO 
= 


Signed administra- 
Tactile graphics/ 
device/software 


in-person 


Beal & Rosenblum 


Hansen et al. 


Pangatungan 
mO} VAN EE (el (-\-m (elma) 
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Table E-3. Equipment Accommodations Itemized by Study 


Author/s 


Electronic ad- 
ministration 
Technological 


TOTAL studies (of 5) 


Author/s 


TOTAL studies (of 4) 


: Ti ae 


Table E-5. Setting Accommodations Itemized by Study 


Author/s 


Individual 


Keiveras | - | 4 


TOTAL studies (of 1) 1 
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